Minimal Residual Approaches for Policy Evaluation in Large Sparse Markov Chains

We consider the problem of policy evaluation in a special class of Markov Decision Processes (MDPs) where the underlying Markov chains are large and sparse. We start from a stationary model equation that the limit of Temporal Difference (TD) learning satisfies, and develop a Robbins-Monro method consistently estimating its coefficients. Then we introduce the minimal residual approaches, which solve an approximate version of the stationary model equation. Incremental Least-squares temporal difference (iLSTD) is shown to be a special form of minimal residual approaches. We also develop a new algorithm called minimal residual (MR) algorithm whose step-size can be computed on line. We introduce the Compressed Sparse Row (CSR) format and reduce the complexity of MR to near that of TD. The advantages of the MR algorithm are that it has comparable data efficiency and computational efficiency to iLSTD, but does not require manual selection of step-size.